Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for platform logs in Filebeat Azure module #22371

Merged
merged 67 commits into from
Nov 25, 2020
Merged

Add support for platform logs in Filebeat Azure module #22371

merged 67 commits into from
Nov 25, 2020

Conversation

narph
Copy link
Contributor

@narph narph commented Nov 3, 2020 •
edited
Loading

What does this PR do?

Add support for platform logs in Filebeat Azure module

Why is it important?

Platform logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Example platform logs:

{
   "ActivityId":"30ed877c-a36b-491a-bd4d-ddd847fe55b8",
   "Caller":"Portal",
   "Environment":"PROD",
   "EventName":"Retreive ConsumerGroup",
   "EventProperties":"{\"SubscriptionId\":\"...\",\"Namespace\":\"obstesteventhubs\",\"Via\":\"sb://obstesteventhubs.servicebus.windows.net/insights-logs-operationallogs/consumergroups?api-version=2017-04\u0026$skip=0\u0026$top=100\",\"TrackingId\":\"30ed877c-a36b-491a-bd4d-ddd847fe55b8_M2CH3_M2CH3_G3S2\"}",
   "EventTimeString":"11/3/2020 9:06:42 AM +00:00",
   "Region":"West Europe",
   "ScaleUnit":"PROD-AM3-AZ501",
   "Status":"Succeeded",
   "SubscriptionId":"....",
   "category":"OperationalLogs",
   "resourceId":"/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS"
}

{
   "Cloud":"AzureCloud",
   "Environment":"prod",
   "UnderlayClass":"hcp-underlay",
   "UnderlayName":"hcp-underlay-westeurope-cx-316",
   "attrs":"{\"annotation.io.kubernetes.container.hash\"=\u003e\"b74d7ef3\", \"annotation.io.kubernetes.container.ports\"=\u003e\"[{\\\"name\\\":\\\"https\\\",\\\"containerPort\\\":4444,\\\"protocol\\\":\\\"TCP\\\"}]\", \"annotation.io.kubernetes.container.preStopHandler\"=\u003e\"{\\\"exec\\\":{\\\"command\\\":[\\\"/bin/bash\\\",\\\"-c\\\",\\\"sleep 20\\\"]}}\", \"annotation.io.kubernetes.container.restartCount\"=\u003e\"0\", \"annotation.io.kubernetes.container.terminationMessagePath\"=\u003e\"/dev/termination-log\", \"annotation.io.kubernetes.container.terminationMessagePolicy\"=\u003e\"File\", \"annotation.io.kubernetes.pod.terminationGracePeriod\"=\u003e\"30\", \"io.kubernetes.container.logpath\"=\u003e\"/var/log/pods/5e4bf4baee195b00017cdbfa_kube-apiserver-666bd4b459-vgc5h_53331907-669d-458e-ab0e-6744e56164f9/kube-apiserver/0.log\", \"io.kubernetes.container.name\"=\u003e\"kube-apiserver\", \"io.kubernetes.docker.type\"=\u003e\"container\", \"io.kubernetes.pod.name\"=\u003e\"kube-apiserver-666bd4b459-vgc5h\", \"io.kubernetes.pod.namespace\"=\u003e\"5e4bf4baee195b00017cdbfa\", \"io.kubernetes.pod.uid\"=\u003e\"53331907-669d-458e-ab0e-6744e56164f9\", \"io.kubernetes.sandbox.id\"=\u003e\"09097fc97b3c0130bde19eec329b588a328430f3c221c1f8e1520933b16dde9f\"}",
   "category":"kube-apiserver",
   "ccpNamespace":"5e4bf4baee195b00017cdbfa",
   "operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
   "properties":{
      "containerID":"ca7ca3b15f428368fabab4dff0c14879a838f8653f84312833d5024547a008f4",
      "log":"I1105 09:40:47.768168       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io\n",
      "pod":"kube-apiserver-666bd4b459-vgc5h",
      "stream":"stderr"
   },
   "resourceId":"/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-INFRASTRUCTURE/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/OBSKUBE",
   "time":"2020-11-05T09:40:47.0000000Z"
}

Ex outputs:

{
        "_index" : "filebeat-8.0.0-2020.11.05-000001",
        "_type" : "_doc",
        "_id" : "YQVzmHUBvB2moownK3Qu",
        "_score" : null,
        "_source" : {
          "agent" : {
            "name" : "DESKTOP-RFOOE09",
            "id" : "17cead09-57ad-4668-8a0e-b9025f8b0cb0",
            "type" : "filebeat",
            "ephemeral_id" : "a9a122d4-4b37-4c34-ac03-4a2f765a9ea1",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "platformlogs"
          },
          "azure-eventhub" : {
            "sequence_number" : 14,
            "consumer_group" : "$Default",
            "offset" : 4294975208,
            "eventhub" : "insights-logs-operationallogs",
            "enqueued_time" : "2020-11-05T10:27:43.439Z"
          },
          "tags" : [
            "forwarded"
          ],
          "cloud" : {
            "provider" : "azure",
            "region" : "West Europe"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-05T10:26:33.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "ingested" : "2020-11-05T12:48:05.927630600Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Retreive Namespace",
            "dataset" : "azure.platformlogs",
            "outcome" : "succeeded"
          },
          "azure" : {
            "subscription_id" : "....",
            "platformlogs" : {
              "Status" : "Succeeded",
              "SubscriptionId" : "....",
              "Caller" : "Portal",
              "ActivityId" : "724eb3a4-6c1d-4a2f-8037-8d4db250154c",
              "EventTimeString" : "11/5/2020 10:26:33 AM +00:00",
              "Environment" : "PROD",
              "category" : "OperationalLogs",
              "event_category" : "Administrative",
              "ScaleUnit" : "PROD-AM3-AZ501",
              "properties" : {
                "SubscriptionId" : "...",
                "TrackingId" : "724eb3a4-6c1d-4a2f-8037-8d4db250154c_M6SN1_M6SN1_G10S1",
                "Namespace" : "obstesteventhubs",
                "Via" : "https://obstesteventhubs.servicebus.windows.net/$Resources/eventhubs?api-version=2017-04&$skip=0&$top=100"
              }
            },
            "resource" : {
              "provider" : "MICROSOFT.EVENTHUB/NAMESPACES",
              "name" : "OBSTESTEVENTHUBS",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS",
              "group" : "OBS-TEST"
            }
          }
        }

{
        "_index" : "filebeat-8.0.0-2020.11.05-000001",
        "_type" : "_doc",
        "_id" : "oQRumHUBvB2moownKezJ",
        "_score" : null,
        "_source" : {
          "agent" : {
            "name" : "DESKTOP-RFOOE09",
            "id" : "17cead09-57ad-4668-8a0e-b9025f8b0cb0",
            "ephemeral_id" : "a6339387-a2fe-4db0-9d13-ca3123f92366",
            "type" : "filebeat",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "platformlogs"
          },
          "message" : """I1105 12:41:49.339404       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
""",
          "azure-eventhub" : {
            "sequence_number" : 45,
            "consumer_group" : "$Default",
            "offset" : 100168,
            "eventhub" : "insights-logs-kube-apiserver",
            "enqueued_time" : "2020-11-05T12:42:14.453Z"
          },
          "tags" : [
            "forwarded"
          ],
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-05T12:41:49.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "ingested" : "2020-11-05T12:42:37.895235200Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
            "dataset" : "azure.platformlogs"
          },
          "azure" : {
            "subscription_id" : "...",
            "platformlogs" : {
              "ccpNamespace" : "5e4bf4baee195b00017cdbfa",
              "operation_name" : "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
              "Cloud" : "AzureCloud",
              "Environment" : "prod",
              "UnderlayClass" : "hcp-underlay",
              "UnderlayName" : "hcp-underlay-westeurope-cx-316",
              "category" : "kube-apiserver",
              "event_category" : "Administrative",
              "properties" : {
                "pod" : "kube-apiserver-666bd4b459-vgc5h",
                "stream" : "stderr",
                "containerID" : "ca7ca3b15f428368fabab4dff0c14879a838f8653f84312833d5024547a008f4"
              },
              "attrs" : """{"annotation.io.kubernetes.container.hash"=>"b74d7ef3", "annotation.io.kubernetes.container.ports"=>"[{\"name\":\"https\",\"containerPort\":4444,\"protocol\":\"TCP\"}]", "annotation.io.kubernetes.container.preStopHandler"=>"{\"exec\":{\"command\":[\"/bin/bash\",\"-c\",\"sleep 20\"]}}", "annotation.io.kubernetes.container.restartCount"=>"0", "annotation.io.kubernetes.container.terminationMessagePath"=>"/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy"=>"File", "annotation.io.kubernetes.pod.terminationGracePeriod"=>"30", "io.kubernetes.container.logpath"=>"/var/log/pods/5e4bf4baee195b00017cdbfa_kube-apiserver-666bd4b459-vgc5h_53331907-669d-458e-ab0e-6744e56164f9/kube-apiserver/0.log", "io.kubernetes.container.name"=>"kube-apiserver", "io.kubernetes.docker.type"=>"container", "io.kubernetes.pod.name"=>"kube-apiserver-666bd4b459-vgc5h", "io.kubernetes.pod.namespace"=>"5e4bf4baee195b00017cdbfa", "io.kubernetes.pod.uid"=>"53331907-669d-458e-ab0e-6744e56164f9", "io.kubernetes.sandbox.id"=>"09097fc97b3c0130bde19eec329b588a328430f3c221c1f8e1520933b16dde9f"}"""
            },
            "resource" : {
              "provider" : "MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS",
              "name" : "OBSKUBE",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-INFRASTRUCTURE/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/OBSKUBE",
              "group" : "OBS-INFRASTRUCTURE"
            }
          }
        },

narph added 30 commits July 23, 2020 11:01
@narph
mofidy doc
938e66c
@narph
Merge branch 'master' of github.com:elastic/beats
4daef08
@narph
Merge branch 'master' of github.com:elastic/beats
b613178
@narph
Merge branch 'master' of github.com:elastic/beats
05364cf
@narph
Merge branch 'master' of github.com:elastic/beats
f147c4d
@narph
Merge branch 'master' of github.com:elastic/beats
4574718
@narph
Merge branch 'master' of github.com:elastic/beats
1e43077
@narph
Merge branch 'master' of github.com:elastic/beats
807cf06
@narph
Merge branch 'master' of github.com:elastic/beats
2096668
@narph
Merge branch 'master' of github.com:elastic/beats
da8ac1f
@narph
Merge branch 'master' of github.com:elastic/beats
c2d8930
@narph
Merge branch 'master' of github.com:elastic/beats
7bd9e73
@narph
Merge branch 'master' of github.com:elastic/beats
6e89a84
@narph
Merge branch 'master' of github.com:elastic/beats
bdf21e9
@narph
Merge branch 'master' of github.com:elastic/beats
7833687
@narph
Merge branch 'master' of github.com:elastic/beats
bbf6178
@narph
Merge branch 'master' of github.com:elastic/beats
4ba8817
@narph
Merge branch 'master' of github.com:elastic/beats
0cba5dc
@narph
Merge branch 'master' of github.com:elastic/beats
b2625ca
@narph
Merge branch 'master' of github.com:elastic/beats
5100e6a
@narph
Merge branch 'master' of github.com:elastic/beats
a302d31
@narph
Merge branch 'master' of github.com:elastic/beats
631d667
@narph
Merge branch 'master' of github.com:elastic/beats
35072a5
@narph
Merge branch 'master' of github.com:elastic/beats
4b2f87a
@narph
Merge branch 'master' of github.com:elastic/beats
f26b533
@narph
Merge branch 'master' of github.com:elastic/beats
c61620d
@narph
Merge branch 'master' of github.com:elastic/beats
43f90c4
@narph
Merge branch 'master' of github.com:elastic/beats
2d28f07
@narph
Merge branch 'master' of github.com:elastic/beats
0a42bbe
@narph
Merge branch 'master' of github.com:elastic/beats
0bedffe
@narph
Copy link
Contributor Author

narph commented Nov 5, 2020

ECS defines the set of values that are allowed to be used in event.category. The azure.platformlogs.category could be mapped to one or more of the allowed values. Like maybe have a table of expected or common azure categories that map to ECS categories (and possibly event.types).

thanks @andrewkroh , looking at the expected types I don't think we could match azure.platform.category to any of the two ECS fields. Here are the possible values we expect https://docs.microsoft.com/en-us/azure/azure-monitor/platform/resource-logs-categories.

@andrewkroh
Copy link
Member

I think there are categories that could be applied. For example there are some database logs in there. And some web activity like for HTTP requests to the data lake (had to look at the data lake schema to check).

@exekias exekias self-requested a review November 9, 2020 09:54
exekias
exekias reviewed Nov 10, 2020
View reviewed changes
x-pack/filebeat/module/azure/platformlogs/test/platformlogs.log-expected.json Outdated
"group" : "OBS-TEST"
},
"platformlogs" : {
"Status" : "Succeeded",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field doesn't have a mapping, right? I see you use them to fill other fields, should it be dropped or mapped?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added mapping for this field

exekias
exekias reviewed Nov 10, 2020
View reviewed changes
x-pack/filebeat/module/azure/platformlogs/test/platformlogs.log Outdated
@@ -0,0 +1 @@
{"ActivityId":"30ed877c-a36b-491a-bd4d-ddd847fe55b8","Caller":"Portal","Environment":"PROD","EventName":"Retreive ConsumerGroup","EventProperties":"{\"SubscriptionId\":\"7657426d-c4c3-44ac-88a2-3b2cd59e6dba\",\"Namespace\":\"obstesteventhubs\",\"Via\":\"sb://obstesteventhubs.servicebus.windows.net/insights-logs-operationallogs/consumergroups?api-version=2017-04\u0026$skip=0\u0026$top=100\",\"TrackingId\":\"30ed877c-a36b-491a-bd4d-ddd847fe55b8_M2CH3_M2CH3_G3S2\"}","EventTimeString":"11/3/2020 9:06:42 AM +00:00","Region":"West Europe","ScaleUnit":"PROD-AM3-AZ501","Status":"Succeeded","SubscriptionId":"7657426d-c4c3-44ac-88a2-3b2cd59e6dba","category":"OperationalLogs","resourceId":"/SUBSCRIPTIONS/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA/RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have several lines here with different types, to give a better idea of what we can expect

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

additional example in

@exekias
Copy link
Contributor

exekias commented Nov 10, 2020

This is looking great! I left a few comments

@narph narph requested a review from exekias November 17, 2020 15:00
@narph narph merged commit 2ae52c3 into elastic:master Nov 25, 2020
@narph narph deleted the platform-logs branch November 25, 2020 10:08
narph added a commit to narph/beats that referenced this pull request Nov 25, 2020
@narph
Add support for platform logs in Filebeat Azure module (elastic#22371)
40ca271 
* mofidy doc

* platformlogs

* fix

* separate pr

* work on platform

* work

* platforms

* changelog

* fix file

* add tests

* add mapping

* test

* update mapping

* fix file name

* update file

* map field

* update files

* fix logs

* generate tests

(cherry picked from commit 2ae52c3)
@narph narph mentioned this pull request Nov 25, 2020
Merged
6 tasks
narph added a commit that referenced this pull request Nov 30, 2020
@narph
Add support for platform logs in Filebeat Azure module (#22371) (#22749)
d950462 
* mofidy doc

* platformlogs

* fix

* separate pr

* work on platform

* work

* platforms

* changelog

* fix file

* add tests

* add mapping

* test

* update mapping

* fix file name

* update file

* map field

* update files

* fix logs

* generate tests

(cherry picked from commit 2ae52c3)
@narph narph added v7.11.0 test-plan Add this PR to be manual test plan and removed [zube]: In Review test-plan Add this PR to be manual test plan labels Jan 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@exekias exekias exekias approved these changes

@leehinman leehinman Awaiting requested review from leehinman

Assignees

@narph narph

Labels
needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team v7.11.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@narph @elasticmachine @andrewkroh @exekias @jsoriano